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APPARATUS AND METHOD FOR MOTION-VECTOR- AIDED INTERPOLATION OF 
A PIXEL OF AN INTERMEDIATE IMAGE OF AN IMAGE SEQUENCE 

5 RELATED APPLICATIONS 

This application is a continuation of application serial number 10/636,298 filed August 7, 

2003. 

BACKGROUND OF THE INVENTION 

10 The present invention relates to the field of video image processing, and in particular to 

an apparatus and a method for motion-vector-aided interpolation of a pixel of an intermediate 
image of an image sequence. 

As known, intermediate image interpolation is required for example in the generation of a 
100 Hz image sequence for visualization in a television set from a received 50 Hz image 

15 sequence, or in the generation of a slow-motion sequence in which a plurality of intermediate 
images are generated in temporal succession between two input images. In order correctly to 
reproduce motion processes in intermediate image interpolation, it is sufficiently well known to 
use motion vectors in intermediate image interpolation, one method for estimating such motion 
vectors is described for example in U.S. Patent 5,386,248. 

20 For better comprehension, the essential procedure for intermediate image interpolation 

using a motion vector is explained below with reference to FIG. 1 . 

FIG. 1 shows schematically a first image and a second image Al, A2 of an input image 
sequence, which are present in temporal succession and to which there is generated an 
interpolated intermediate image A 12 that, in correspondence to the input images Al, A2, has a 
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plurality of pixels of which one pixel Px is schematically illustrated in FIG 1. Assigned to this 
pixel is at least one motion vector vecl, which is generated on the basis of the input image 
sequence. The generation of the motion vector is based on the fact that a moving object B is 
located at a first position in the first input image Al at the time of the first input image, and 
"shifts" to a second position in the second input image A2 by the time the second input image is 
"photographed". The motion vector vecl contains the motion information of this object. Along 
with the motion vector vecl, the position of the object after the motion is depicted in the input 
image Al and the position of the object before the motion is depicted in the input image A2 for 
clarification. Using the motion vector vecl and the video information value PO at the initial point 
of the vector vecl in the image Al, or the video information value PI at the final point of the 
motion vector vecl in the image A2, and with allowance for the raster position of the 
intermediate image A12 relative to the input images Al and, A2, the pixel Px of the intermediate 
image lying on the motion line of the vector vecl can be determined. 

Selecting the pixel PO lying at the initial point of the motion vector vecl assigned to the 
pixel Px in the first image or the pixel PI lying at the final point of the motion vector vecl in the 
image A2 is theoretically sufficient for the interpolation of the pixel Px. 

However, the estimation of the motion vector or motion vectors, involves uncertainties. 
For this reason it is known to assign two motion vectors to a pixel to be interpolated, to apply 
median filtering to the video information item of the initial point in a first input image and the 
video information item of the final point in a second input image of the one motion vector as 
well as the average of the video information items of the initial point and the final point of the 
other motion vector, and to use the video information value resulting therefrom as the video 
information value for the pixel to be interpolated. 



An object of the present invention is to furnish an improved method for motion- vector- 
aided intermediate image interpolation. 

SUMMARY OF THE INVENTION 

The motion-vector-aided interpolation of a pixel of an intermediate image lying between 
two input images includes selecting from the first input image a first pixel to which a first video 
information value is assigned, using a first motion vector, and selecting from the second input 
image a second pixel to which a second video information value is assigned, using the first 
motion vector. A third pixel to which a third video information value is assigned is selected from 
the first input image using a second motion vector, and a fourth pixel to which a fourth video 
information value is assigned is selected from the second input image using the second motion 
vector. The determination of the motion vectors that are assigned to the pixel to be interpolated 
can be determined by conventional methods for the estimation of motion vectors. 

After the first to fourth video information values have been determined, an interval 
specified by the first video information value and the second video information value is 
determined and/or an interval specified by the third video information value and the fourth video 
information value is determined. The video information values are mixed in that the first video 
information value is multiplied by a first weighting factor, the second video information value by 
a second weighting factor, the third video information value by a third weighting factor and the 
fourth video information value by a fourth weighting factor, and the weighted video information 
values so obtained are added to obtain a video information value of the pixel of the intermediate 
image. The weighting factors are selected such that this video information value lies within the 



interval specified by the first video information value and the second video information value or 
the interval specified by the third video information value and the fourth video information 
value. 

Preferably, the interval specified by the first video information value and the second 
video information value is determined and the interval specified by the third video information 
value and the fourth video information value is determined, the interval whose span between 
interval bounds is smaller in absolute value being used in the determination of the weighting 
factors. 

In one embodiment, provision is made for equally weighting the first video information 
value and the second video information value and/or equally weighting each of the third and the 
fourth video information value. 

In a further embodiment, provision is made for selecting a zero vector as one of the two 
motion vectors. By the choice of one of the two motion vectors as a zero vector, the video 
information items of the pixels located in the first input image and the second input image at the 
same position as the pixel to be interpolated flow into the interpolation of the pixel of the 
intermediate image. 

The interpolation preferably takes place in steps, a first intermediate value first being 
generated by mixing of the first video information value and the second video information value, 
and a second intermediate value being generated by mixing of the third video information value 
and the fourth video information value and the resultant intermediate values being weighted, 
using a further weighting factor in order to obtain the video information value of the pixel of the 
intermediate image. The first video information value and the second video information value are 
preferably equally weighted in the generation of the first intermediate value, with the overall 



result of equal weighting of the first video information value and the second video information 
value at the pixel to be interpolated. Furthermore, the third video information value and the 
fourth video information value are equally weighted in the generation of the second intermediate 
value, with the overall result of equal weighting of the third video information value and the 
fourth video information value at the pixel to be interpolated. 

These and other objects, features and advantages of the present invention will become 
more apparent in light of the following detailed description of preferred embodiments thereof, as 
illustrated in the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWING 

FIG. 1 is a schematic depiction of two input images and an intermediate image 
interpolated using a motion vector; 

FIG. 2 is a schematic depiction of two input images and an intermediate image 
interpolated using two motion vectors; 

FIG. 3 is a block diagram illustration of a first embodiment of a vector aided interpolation 

device; 

FIG. 4 is a block diagram illustration of a second embodiment of a vector aided 
interpolation device; 

FIG. 5 is a block diagram illustration of a third embodiment of a vector aided 
interpolation device; 

FIG. 6 is a block diagram illustration of a fourth embodiment of a vector aided 
interpolation device; and 

FIG. 7 is a block diagram illustration of a fifth embodiment of a vector aided 
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interpolation device. 

DETAILED DESCRIPTION OF THE INVENTION 

The interpolation of a pixel Px of an intermediate image A12 depicted schematically in 
5 FIG. 2 for two input images Al, A2 includes the assignment of a first motion vector and a 
second motion vector vecl, vec2 to the pixel Px to be interpolated. The motion vectors may be 
determined by conventional methods for the estimation of motion vectors. On the basis of the 
first motion vector, a first pixel PO is determined in the first input image Al, which pixel 
corresponds to the initial point of the first motion vector vecl passing through the pixel Px to be 

10 determined in the first input image Al. Furthermore, on the basis of the first motion vector, a 
second pixel PI is determined in the second input image A2, which motion vector corresponds to 
the final point of the first motion vector vecl. In a corresponding fashion, using the second 
motion vector vec2 passing through the pixel Px to be interpolated, a third pixel P2 is determined 
in the first input image, wherein the pixel corresponds to the initial point of the motion vector 

15 vec2, and a fourth pixel P3 is determined in the second input image, wherein the pixel 
corresponds to the final point of the motion vector vec2. 

Assigned to each of these pixels PO, PI, P2, P3 are video information values L0, LI, L2, 
L3, respectively, for example brightness values or luminance values, color information items, 
contrast information items or other processable representatives, which video information values 

20 are used for the interpolation of a corresponding video information value of the pixel Px to be 
interpolated. 

An aspect of the invention includes determining an interval specified by the first video 
information value L0 and the second video information value LI and mixing the video 
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information values L0-L3 by multiplying the first video information value LO by a first 
weighting factor kO, the second video information value LI by a second weighting factor kl, the 
third video information value L2 by a third weighting factor k2 and the fourth video information 
value L3 by a fourth weighting factor k3 and adding the weighted video information values in 
order to obtain the video information value Lx of the interpolated pixel Px. 

The weighting factors are generated such that the video information value Lx of the 
interpolated pixel Px lies within the interval specified by the first video information value and 
the second video information value LO, LI . Thus: 

Lx = LO-kO + Ll-kl +L2-k2 + L3-k3 (1) 

and 

Lx is an element of the interval [L0;L1] (2) 

the video information value Lx preferably not lying at the interval bounds, that is, not assuming 
the values LO or LI. 

In an embodiment, provision is made for determining an interval specified by the third 
video information value and the fourth video information value and, in the determination of the 
weighting factors kO, kl, k2, k3 with the above inequality (1), (2), using the interval whose 
difference between interval bounds is smaller in absolute value. Thus if |L2-L3| < |L0-L1|, the 
weighting factors kO, kl, k2, k3 and the video information value Lx are determined with 
Equation (1) and the following relation: 



Lx is an element of [L2-L3] 



(3) 



In the determination of the choice of weighting factors, several degrees of freedom exist, 
the constraint being: 



kO + kl + k2 + k3 - 1 (4) 
and for each of the weighting factors: 

0<ky< 1, where y = 0,1,2,3 (5) 

FIG. 3 is a block diagram illustration of a first embodiment of a vector aided interpolation 
device. The apparatus comprises a selector circuit 10, that receives a video signal s(t) and 
prepares the video information values L0, LI, L2, L3, which are generated using a first motion 
vector and a second motion vector vecl, vec2, which are generated in the selector circuit or 
supplied externally. The video information values are supplied to an analyzer 20, which prepares 
the weighting factors kO, kl, k2, k3, taking account of Equations (1) and (2) or (1) and (3). 
Together with the video information values L0, LI, L2, L3, these weighting factors kO, kl, k2, 
k3 are supplied to a mixer 30, which effects mixing according to Equation (1) in order to prepare 
the video information value Lx of the interpolated pixel Px. FIG. 4 shows a second embodiment 
of a vector aided interpolation device. 

FIG. 4 is a block diagram illustration of a second embodiment of a vector aided 
interpolation device. The mixing of the video information values L0-L3 may take place in steps, 
a first intermediate value M01 first being generated in a mixer 32 from the first video 



information value and the second video information value LO, LI, using a mixing factor kOl, 
according to the relation: 



M01=k01-L0 + (l-k01)-Ll (6) 

A second intermediate value M23 is generated in a second mixer 33 from the third video 
information value and the fourth video information value L2, L3, using a mixing factor k23, 
according to the relation: 

M23 = k23-L2 + (l-k23) -L3 (7) 
The following holds for the weighting factors kOl, k23: 

kOl + k23 = 1 ( 8 ) 

and 

0<k01 < 1 andO< k23 < 1 (9) 

The first intermediate value M01 and the second intermediate value M23 are supplied to 
a third mixer 34, which prepares the video information value Lx from the first intermediate value 
and the second intermediate value M01, M23, using a further mixing factor k0123 prepared by 
the analyzer 22 and using the following equation: 

Lx = (l-k0123)-M01 + k0123-M02 (9) 
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where 0< k0123< 1. 



In a further embodiment illustrated in FIG. 5, provision is made for holding constant the 
weighting factors kOl and k23 for the generation of the first intermediate value and the second 
5 intermediate value M01, M23 respectively, and choosing for example kOl = k23 = 1/2. In this 
case the first video information value LO and the second video information value LI are equally 
weighted in the video information value of the interpolated pixel and the third video information 
value L2 and the fourth video information value L3 are equally weighted in the video 
information value Lx of the interpolated pixel, independently of the weighting factor k0123. The 
10 analyzer 24 in this case furnishes only the weighting factor k0123 from the video information 
values LO, LI, L2, L3, the following holding for kOl - k23 = 1/2: 

Lx =(l-k0123)-M01+(k0123-M02)= 

( 1 -kO 1 23> 1 /2-(L0+L 1 ) + kO 1 23 ■ 1 /2-(L0+L 1 ) (10) 

15 

Here Lx lies within the interval specified by [L0;L1] if the absolute value of the difference 
between Lx and the midpoint of the interval [L0;L1] is less than half the interval width or 
corresponds to half the interval width, that is, if the following holds: 

20 |Lx-(L0+Ll)/2|<|L0-Ll|/2 (11) 

If Equation (10) is substituted in Equation (11), k0123 must satisfy the following 
inequality if the video information value Lx of the pixel to be interpolated is to lie within this 
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interval: 



k0123 < |L0-L1|/|(L2+L3) - (L0+L1) | (12) 

the analyzer selects the weighting factor k0123 according to this inequality. 

If the second intermediate value M23 already lies within the interval [L0,L1] specified by 
the video information values LO, LI, that is, if: 

2-(|M01| - |M23|) < |L0-L1| or 2<|L0+Ll|/2 - |L2+L3|/2) < |L0-L1| (13) 

then k0123 can be chosen arbitrarily from values between 0 and 1, that is, from the interval [0,1]. 

If M23 lies outside this interval, k0123 must satisfy inequality (12), k0123 preferably 
being chosen such that the video information value Lx does not lie at one of the interval bounds, 
that is, does not correspond to the video information value LO or LI. Preferably, k0123 is 
selected such that the resulting video information value Lx maintains a specified distance relative 
to the interval bounds, a condition that is satisfied if: 

k0123 < |L0-Ll|/a-|(L2+L3) - (L0+L1) | (14) 
for all a> L 

The factor a is preferably greater than two (2). 

FIG. 6 shows an apparatus for implementing such a method, the apparatus having an 
analyzer 26 to which the video information values L0-L3 as well as the intermediate values M01, 
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M23 obtained by equal weighting of the video information values L0-L3 are supplied for the 
determination of the weighting factor k0123. 

In a further embodiment of the method according to the invention, provision is made for 
outputting either the first intermediate value M01 or the second intermediate value M23 as the 
video information value Lx. FIG. 7 shows an apparatus for implementing such a method, a 
switch S being provided in a mixer 35 connected subsequently to the mixers 32, 33, the switch 
being thrown depending on the weighting factor k0123. In this embodiment the weighting factor 
k0123 assumes the value 0 or 1, the first intermediate value M01 yielding the video information 
value Lx for a value of 0 and the second intermediate value yielding the video information value 
Lx of the interpolated pixel for a value of 1 of the weighting factor k0123. 

The weighting factor k0123 is determined depending on the video information values LO, 
LI, L2, L3, the video information value being set equal to one (1), for example, in order to select 
the second intermediate value if the second intermediate value M23 lies within the interval 
specified by [L0;L1], and being set equal to zero in order to select the first intermediate value 
M01 if the second intermediate value M23 lies outside the interval specified by [L0;L1]. 

In an embodiment of the invention, provision is made for choosing one of the two motion 
vectors vecl, vec2 as a zero vector. This ensures that the video information value of the pixel of 
the first input image Al whose position corresponds to the position of the pixel Px to be 
interpolated and the video information value of the pixel of the second input image A2 whose 
position corresponds to the position of the pixel Px to be interpolated are taken into account in 
the determination of the video information value Lx of the interpolated pixel. 

In a further embodiment, provision is made for generating a first interpolated video 
information value in one of the manners explained above, using the first interval specified by the 
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first video information value and the second video information value, and generating a second 
interpolated video information value, using the second interval specified by the third video 
information value and the fourth video information value, in order to interpolate the video 
information value. These two video information values are then mixed together to form the 
video information value Lx, for example by taking the average. 

Although the present invention has been shown and described with respect to several 
preferred embodiments thereof, various changes, omissions and additions to the form and detail 
thereof, may be made therein, without departing from the spirit and scope of the invention. 

What is claimed is: 
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